Ranking documents using Large Language Models (LLMs) by directly feeding the query and candidate documents into the prompt is an interesting and practical problem. However, there has been limited success so far, as researchers have found it difficult to outperform fine-tuned baseline rankers on benchmark datasets. We analyze pointwise and listwise ranking prompts used by existing methods and argue that off-the-shelf LLMs do not fully understand these ranking formulations, possibly due to the nature of how LLMs are trained. In this paper, we propose to significantly reduce the burden on LLMs by using a new technique called Pairwise Ranking Prompting (PRP). Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs. On TREC-DL2020, PRP based on the Flan-UL2 model with 20B parameters outperforms the previous best approach in the literature, which is based on the blackbox commercial GPT-4 that has 50x (estimated) model size, by over 5% at NDCG@1. On TREC-DL2019, PRP is only inferior to the GPT-4 solution on the NDCG@5 and NDCG@10 metrics, while outperforming other existing solutions, such as InstructGPT which has 175B parameters, by over 10% for nearly all ranking metrics. Furthermore, we propose several variants of PRP to improve efficiency and show that it is possible to achieve competitive results even with linear complexity. We also discuss other benefits of PRP, such as supporting both generation and scoring LLM APIs, as well as being insensitive to input ordering.

Last fourth of July, as fireworks burst across the night sky near the Lieber Correctional Institution in Ridgeville, S.C., convicted kidnapper Jimmy Causey tucked a lifelike dummy into his bed, sneaked out of his prison cell and completed a daring escape. It wasn't until three days later, when Texas Rangers found Causey holed up 1,200 miles away, that authorities offered an explanation for how he had obtained the equipment for the breakout, including a pair of wire cutters used to snip through four fences that encircle the maximum security prison. "We believe a drone was used to fly in the tools that allowed him to escape," Bryan Stirling, director of the South Carolina Department of Corrections, told reporters at a news conference. A lengthy investigation confirmed that an accessory role was played by a small, off-the-shelf drone. And with that, law-enforcement and national security officials added "prison breaks" to the potential ill uses lurking in a technology widely available at retailers including Amazon and Walmart.